Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during fusion, generated may be redundant, and discriminative unimodal information ignored, which often interferes with accurate prediction leads to higher risk overfitting. Moreover, representations also contain noisy negatively influences learning dynamics. To this end, we introduce bottleneck (MIB), aiming learn powerful sufficient representation is free redundancy filter out representations. Specifically, inheriting from general (IB), MIB aims minimal given task by maximizing mutual between target simultaneously constraining input data. Different IB, our regularizes both representations, comprehensive flexible framework compatible any fusion methods. develop three variants, namely, early-fusion MIB, late-fusion complete on different perspectives constraints. Experimental results suggest proposed method reaches state-of-the-art performance tasks sentiment analysis emotion recognition across widely used datasets. The codes are available at https://github.com/TmacMai/Multimodal-Information-Bottleneck.